TidyText analysis

library(dplyr)

Registered S3 method overwritten by 'dplyr':
  method           from
  print.rowwise_df     

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

glimpse(movies_dataset)

Observations: 2,000
Variables: 2
$ class <chr> "Pos", "Pos", "Pos", "Pos", "Pos", "Pos", "Pos", "Pos", "Pos", …
$ text  <chr> "films adapted from comic books have had plenty of success   wh…

Bag of Words Tokenisation

In this approach, we represent each word in a document as a token (or feature) and each document as a vector of features. In addition, for simplicity, we disregard word order and focus only on the number of occurrences of each word i.e., we represent each document as a multi-set ‘bag’ of words.

dtm<-movies_dataset %>% select(-class)   %>% 
  mutate(row=row_number())  

dtm <- dtm %>% unnest_tokens(word,text) %>% group_by(word,row) %>% summarise(total=n()) %>% cast_sparse(row,word,total)

dtm
str(as.matrix(dtm))
as.matrix(dtm)[1:2,2000:2030] 

Remove Stop Words

i

Term frequency

The statistic tf-idf is intended to measure how important a word is to a document in a collection (or corpus) of documents, for example, to one novel in a collection of novels or to one website in a collection of websites.

dtm<-movies_dataset %>% select(-class)  %>% sample_n(100) %>% 
  mutate(row=row_number()) %>% unnest_tokens(word,text) %>% group_by(word,row) %>% 
  summarise(total=n()) %>%
  anti_join(stop_words)
row_words <- dtm%>% count(row,word, sort=TRUE)
total_words <- dtm %>% group_by(row) %>% summarise(total=n())

inner_join(row_words,total_words) %>%   bind_tf_idf(word, row, n)